Merge scan options into FileIO at TableScan.__init__ time#3258
Merge scan options into FileIO at TableScan.__init__ time#3258rccreager wants to merge 1 commit intoapache:mainfrom
Conversation
|
I have mixed feelings about recreating FileIO here. Even if this works in practice, creating a new IO instance during I also tried to find a similar pattern in the Java implementation and could not find one. There, scan options are added to scan context, while scan planning still uses table.io():
From what I could find, scan options there are mainly used for scan planning overrides like split sizing / lookback / open-file-cost, not for reconfiguring FileIO:
|
2 participants
Closes #3166
Rationale for this change
The
Table.scan(..)method accepts a parameteroptions: Properties; this parameter is not actually threaded into the FileIO instance used in the DataScan class which scans the data. Instead, the internal callers were using a FileIO instance with only the basic catalog-level parameters set and were not respecting the user-specified options.This is resolved by merging scan
optionsinto the FileIO atTableScan.__init__time, using the existingload_file_iomethod. This means that if the user provides new parameters toscan, a new FileIO instance must be created with the merged properties. FileIO properties are immutable after construction, and the cached filesystem objects are built from those properties at first access, so a new instance is required.If the
optionsare empty or if the key:value pairs are all present in the existing FileIO options, you can use the existing FileIO instance.I wanted to fix the problem at initialization, rather than at the various callsites, because
self.iois also used inscan_plan_helper()for manifest reads, not just into_arrow()for data reads. Fixing at init time covers both.Are these changes tested?
I added tests to confirm that the init method has the expected behavior.
Are there any user-facing changes?
Yes. Scan-level options passed to Table.scan(options=...) are now respected by the underlying FileIO during data and manifest reads. Previously these options were silently ignored.